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PREFACE 


The  research  documented  in  this  technical  report  for  the  Battlespace 
Logistics  Readiness  and  Sustainment  program  was  sponsored  by  the  Air 
Force  Research  Laboratory,  Human  Effectiveness  Directorate,  Logistics 
Readiness  Branch  (AFRL/HEAL),  Wright-Patterson  AFB,  OH.  The  research 
was  conducted  under  Delivery  Order  #26  of  the  Technology  for  Readiness 
and  Sustainment  (TRS)  contract  (F33615-99-D-6001).  The  University  of 
Arkansas,  Department  of  Industrial  Engineering  performed  the  work  with  the 
assistance  of  Northrop  Grumman  Information  Technology.  Edward  S.  Boyle 
(AFRL/HEAL)  was  the  program  manager  for  this  effort. 
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1.  Introduction 


The  Battlespace  Logistics  Readiness  &  Sustainment  Research  was  an  effort  initiated  by 
the  Air  Force  Research  Laboratory’s  Logistics  Readiness  Branch  (AFRL/HEAL)  to  provide 
specialized  research  to  develop  and  evaluate  logistics  technologies.  Within  the  scope  of  this 
program  specific  research  tasks  could  be  focused  on  feasibility  studies,  cost  benefit  analyses, 
modeling  and  simulation  data  and  algorithms,  front-end  analyses,  field  test  support  activities,  and 
demonstration  system  development  within  the  domains  of  operational,  contingency,  and 
acquisition  logistics.  Operational  logistics  emphasizes  improving  the  performance  of  logistics 
personnel  in  all  operational  environments.  Contingency  logistics  encompasses  technologies  to 
improve  the  speed,  efficiency  and  ease  of  deployment  of  all  logistics  support  elements  necessary 
to  support  contingency  operations.  Acquisition  logistics  focuses  on  improving  the  logistics 
elements  of  systems  during  development  through  improved  system  design  support  and 
information  technologies. 

After  reviewing  a  wide  range  of  logistics  challenges  and  research  requirements,  and 
assessing  the  budget  realities,  AFRL/HEAL  decided  to  narrow  the  actual  research  that  would  be 
conducted  to  the  five  areas  of  interest  listed  below. 

•  Modeling  Sortie  Generation  for  Unit-Level  Logistics  Planners 

•  Maintenance  Decision-Making  under  Prognostic  and  Diagnostic  Uncertainty 

•  Quantifying  the  Impacts  of  Improvements  to  Prognostic  and  Diagnostic 
Capabilities 

•  Multi-State  Selective  Maintenance  Decisions 

•  Quantification  of  Logistics  Capabilities 

Tasks  covering  each  of  those  five  areas  were  developed  and  packaged  as  a  delivery  order  that 
was  performed  over  an  18  month  period  by  the  University  of  Arkansas,  Department  of  Industrial 
Engineering,  with  the  assistance  of  Northrop  Grumman,  under  AFRL’s  Technology  for 
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Readiness  and  Sustainment  (TRS)  research  contract  with  Northrop  Grumman.  Individual  reports 
were  written  detailing  the  research  conducted  on  each  task. 

2.  Modeling  Sortie  Generation  for  Unit-Level  Logistics  Planners 

This  research  demonstrated  that  simulation  could  provide  valuable  information  for 
decision-making  at  the  unit-level  and  provide  much  needed  assistance  in  the  generation  and 
execution  of  a  weekly  flying  schedule.  In  research  performed  by  the  University  of  Arkansas  on  a 
previous  TRS  delivery  order,  a  simulation  model  detailing  an  Air  Force  Multi-Indenture  Multi- 
Echelon  (MIME)  [McGee,  2004]  repairable  parts  system  was  developed.  That  simulation  model 
was  used  to  explore  the  effects  of  using  commercial  shipping  practices  on  the  supply  chain.  This 
research  expanded  the  simulation  model  developed  for  the  commercial  transportation  project  to 
detail  the  sortie  generation  process.  Using  this  simulation  the  effective  risk  inherent  in  a  given 
weekly  schedule  was  evaluated.  The  concept  was  to  allow  until-level  logistics  planners  to  input 
a  weekly  schedule  and  to  evaluate  that  schedule  based  on  outputs  received  from  the  simulation. 
A  model  of  this  type  also  allows  the  unit-level  logistics  planners  to  compare  alternative 
schedules  and  perform  what-if  analysis. 

There  are  significant  challenges  in  developing  a  tool  that  is  useful  at  the  unit-level.  First, 
the  user  interface  must  be  such  that  the  planners  can  quickly  and  easily  input  the  needed  data.  If 
the  user  interface  is  too  complex  or  confusing  no  value  will  be  added  to  the  decision  process. 
Second,  the  simulation  must  be  detailed  enough  to  provide  useful  outputs  to  the  user,  but  also 
simple  enough  that  the  simulation  time  and  data  requirements  are  minimal. 

This  research  began  by  reviewing  the  relevant  literature,  systems,  and  processes.  Part  of  the  goal 
was  to  illustrate  how  simulation  could  be  used  in  this  context.  Thus,  military  logistics,  MIME 


2 


systems,  and  user  interface  systems  required  review.  Likewise,  the  project  provided  a  detailed 
description  of  the  simulation  model  along  with  the  user  interface  system  and  a  description  of  data 
handling  methods.  In  addition,  a  test  scenario  was  outlined  along  with  detail  on  how  the  model 
handles  this  scenario  from  user  input  to  interpreting  the  output  data.  Examples  of  what-if 
analysis  that  can  be  performed  using  the  same  test  scenario  were  also  developed.  Through  the 
execution  of  this  test  scenario,  the  research  demonstrated  how  this  type  of  simulation  technology 
could  be  useful  in  making  decisions  at  the  unit-level.  The  outputs  will  provide  the  user  valuable 
assistance  in  making  timely  decisions  that  can  mitigate  the  risk  associated  with  executing  a 
flying  schedule. 

3.  Maintenance  Decision-Making  under  Prognostic  and  Diagnostic  Uncertainty 

A  key  challenge  faced  by  USAF  maintenance  personnel  is  the  uncertainty  associated  with 
the  information  provided  by  many  of  their  diagnostic  tools.  This  uncertainty  results  from 
precision  and  accuracy  issues  associated  with  individual  diagnostic  tools,  as  well  as 
inconsistencies  between  different  diagnostic  tools  (such  as  different  testing  parameters  embedded 
in  aircraft  built-in-test  (BIT)  systems  for  a  component  versus  the  parameters  in  the  test  sets  used 
by  flightline  and  shop  technicians  to  troubleshoot  that  same  component).  This  uncertainty  can 
make  it  very  difficult  for  maintenance  technicians  to  choose  an  appropriate  course  of  action.  The 
end  result  is  the  possible  omission  of  necessary  maintenance  actions  and/or  the  performance  of 
unnecessary  actions.  Both  of  these  potential  mistakes  can  cause  delays  in  returning  an  aircraft  to 
mission  capable  status  and  increased  requirements  for  spare  parts  in  the  supply  chain. 

This  research  developed  a  methodology  based  on  mathematical  modeling  that  can  be 
used  to  synthesize  the  diagnostic  information  and  provide  a  recommended  course  of  action  to  the 
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technician.  Ultimately,  this  methodology  could  potentially  be  incorporated  into  a  decision- 
support  tool  for  the  technician. 

This  research  was  conducted  using  a  hypothetical  weapon  system;  however,  that 
hypothetical  weapon  system  was  defined  such  that  it  possesses  fundamental  characteristics 
similar  to  systems  utilized  by  the  U.S.  Air  Force.  Modeling-based  methodologies  were 
developed  for  synthesizing  diagnostic  information  and  providing  an  estimated  assessment  of  the 
system.  A  Bayesian  probability  approach  was  defined  for  synthesizing  imperfect  and  conflicting 
diagnostic  information,  along  with  the  characteristics  of  the  system  of  interest  and  the 
diagnostics  applied  to  that  system. 

The  research  demonstrated  how  a  Bayesian  analysis  could  be  used  to  provide  an 
assessment  of  system  status,  and  using  a  numerical  example,  demonstrated  the  potential 
effectiveness  of  the  approach.  First,  the  system  structure  and  the  reliability  and  maintainability 
characteristics  of  each  component  in  the  system  were  defined.  Second,  the  characteristics  of  the 
diagnostic  tools  applied  to  the  system  were  identified.  This  identification  included  a  description 
of  the  precision  and  accuracy  in  diagnostic  information.  Third,  a  set  of  mathematical  and  logical 
models  were  developed  which  synthesized  the  diagnostic  information  and  provided  an  estimated 
assessment  of  the  system.  Finally,  numerical  experiments  were  utilized  for  assessing  the 
capabilities  of  the  defined  models. 

The  Bayesian  approach  showed  great  promise  as  a  means  of  compiling  imperfect  and 
conflicting  diagnostic  information;  however,  that  approach  requires  exact  monitoring  of 
component  aging  and  perfect  life  distribution  estimation.  Furthermore,  that  approach  requires  an 
assumption  of  independent  component  failures.  Therefore,  an  alternative  approach  based  on 
artificial  neural  networks  (ANN)  was  explored.  This  approach  does  not  suffer  from  either  of  the 
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identified  limitations  of  the  Bayesian  approach;  however,  the  numerical  results  associated  with 
this  approach  were  not  as  promising.  The  specifics  of  the  research  and  the  methodology  used, 
along  with  the  mathematical  calculations,  are  detailed  in  a  separate  project  report. 

4.  Quantifying  the  Impacts  of  Improvements  to  Prognostic  and  Diagnostic  Capabilities 

A  key  challenge  faced  by  USAF  maintenance  personnel  is  the  imperfection  of  aircraft 
diagnostic  and  prognostic  capabilities.  These  imperfections  include  an  inability  to  pinpoint  failed 
components,  the  incorrect  identification  of  failed  components,  and  “Could  Not  Duplicate 
(CND)”  or  “Retest  Okay  (RETOK)”  errors  between  flightline  and  shop  tests  and  between  base- 
level  and  depot  tests.  These  imperfections  lead,  among  other  things,  to  increased  delays  in 
returning  aircraft  to  mission  capable  status  and  to  excessive  spare  parts  requirements. 
Unfortunately,  it  is  difficult  to  assess  the  impact  of  improvements  to  diagnostic  and  prognostic 
capabilities. 

This  research  developed  a  methodology  based  on  mathematical  modeling  for  analyzing 
those  impacts.  Specifically,  the  following  questions  were  addressed:  (1)  What  impact  does 
diagnostic  and  prognostic  errors  have  on  fleet  readiness  and  the  associated  requirements  for 
spare  parts  investment?  (2)  Given  a  specific  investment  in  diagnostic  and  prognostic 
improvements,  what  will  the  impact  be  on  fleet  readiness  and  spare  parts  inventory  measures? 
(3)  Given  a  limited  budget  for  diagnostic  and  prognostic  improvements,  how  should  the  funds  be 
allocated  to  optimize  fleet  readiness  and  spare  parts  inventory  measures? 

As  in  the  Maintenance  Decision-Making  under  Prognostic  and  Diagnostic  Uncertainty 
research  discussed  above,  this  research  utilized  hypothetical  systems  that  possessed  key 
characteristics  similar  to  those  utilized  by  the  U.S.  Air  Force.  In  the  first  phase  of  the  project,  a 
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set  of  mathematical  and  logical  modeling  tools  related  to  diagnostic  errors  was  created.  A 
modeling  framework  for  a  single  system  was  established,  and  simulation  modeling  efforts  were 
described  for  evaluating  the  readiness  of  a  single  system  and  a  set  of  systems  under  diagnostic 
errors.  In  addition,  an  optimization  model  for  allocation  of  a  diagnostic  improvement  budget  was 
presented. 

In  the  second  phase  of  this  research,  a  simulation  modeling  environment  to  compare 
prognostics  to  scheduled  maintenance  was  used.  As  a  first  step,  the  system  structure  and  the 
reliability  and  maintainability  characteristics  of  each  component  in  the  system  were  defined. 
Second,  the  characteristics  of  the  diagnostic  and  prognostic  tools  applied  to  the  system  were 
identified.  This  identification  included  a  precise  description  of  the  potential  for  imperfections  in 
diagnostic  and  prognostic  information.  Third,  a  set  of  mathematical  and  logical  models  was 
developed  that  could  be  used  to  measure  the  performance  of  the  fleet  with  and  without  diagnostic 
and  prognostic  errors.  This  permitted  assessment  of  the  impact  of  diagnostic  and  prognostic 
imperfections  on  fleet  readiness  and  spare  parts  inventory  investment.  Specifically,  it  was  shown 
that  prognostics  could  be  an  effective  tool  in  some  cases  (even  in  the  presence  of  significant 
prognostics  errors)  and  a  very  ineffective  tool  in  other  cases.  In  some  cases,  prognostic  errors 
could  make  a  situation  worse  than  if  the  system  were  simply  run  until  it  failed.  Fourth,  this 
model  was  used  to  explore  the  impact  of  specific  investments  in  diagnostic  and  prognostic  tools. 
That  permitted  evaluation  of  the  cost-effectiveness  of  potential  diagnostic  and  prognostic 
improvement  actions.  Finally,  the  model  was  incorporated  into  a  decision-support  environment 
designed  to  allocate  investments  in  diagnostic  and  prognostic  improvements.  That  permitted  the 
decision-maker  to  fund  cost-effective  diagnostic  and  prognostic  improvement  efforts  that 
optimized  fleet  performance. 
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5.  Multi-State  Selective  Maintenance  Decisions 


All  military  organizations  depend  on  the  reliable  performance  of  repairable  systems  for 

the  successful  completion  of  missions.  The  use  of  mathematical  modeling  for  the  purpose  of 

modeling  repairable  systems  and  designing  optimal  maintenance  policies  for  those  systems  has 

received  an  extensive  amount  of  attention  in  the  literature.  Unfortunately,  the  vast  majority  of 

that  work  ignores  potential  limitations  on  the  resources  required  to  perform  maintenance  actions. 

That  shortcoming  has  motivated  the  development  of  models  for  selective  maintenance,  the 

process  of  identifying  the  subset  of  actions  to  perform  from  a  set  of  desirable  maintenance 

actions.  Previously,  researchers  have  developed  a  class  of  mathematical  models  that  can  be  used 

to  identify  selective  maintenance  decisions  for  the  following  scenario: 

A  weapon  system  has  just  completed  a  mission  and  will 
begin  its  next  mission  soon.  Maintenance  cannot  be 
performed  duhng  missions;  therefore,  the  decision-maker 
must  decide  which  components  to  maintain  prior  to  the  next 
mission. 

The  selective  maintenance  models  considered  to  date  treat  decision-making  for  binary-state 
systems — all  components,  subsystems,  and  the  system  itself  are  assumed  to  be  either  functioning 
or  failed  at  any  point  in  time.  As  a  result,  mission  reliability  was  used  as  the  objective  function 
in  the  resulting  mathematical  programming  models. 

This  research  improved  upon  that  approach  in  two  ways.  First,  it  is  more  realistic  to 
classify  component  status  using  more  than  two  discrete  levels  (if  not  some  continuous  measure). 
This  implies  multi-state  measures  of  subsystem  and  system  status  as  well.  Thus,  this  research 
developed  a  modeling-based  methodology  for  managing  selective  maintenance  decisions  when 
multiple  (more  than  two)  system  states  are  possible.  Second,  the  performance  of  a  military 
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system  typically  can  be  measured  using  several  measures  in  addition  to  mission  reliability.  All 
these  performance  measures  are  functions  of  the  status  of  the  components.  This  research 
developed  multi-state  selective  maintenance  models  that  incorporate  multi-state  component 
status  and  multiple  measures  of  system  performance. 

A  scenario  was  defined  in  which  systems  in  various  states  of  maintenance  need  must 
perform  a  number  of  different  missions.  First,  the  system  structure  and  appropriate  status 
measures  were  defined  for  each  component  in  the  system.  Second,  the  resources  consumed  by 
maintenance  actions,  the  impact  on  component  status  of  each  potential  maintenance  action,  and 
the  quantity  of  each  resource  consumed  by  each  maintenance  action  were  identified.  Third,  the 
relevant  measures  of  mission  performance  were  identified  and  functions  were  developed  that 
captured  those  measures  in  terms  of  the  component  status  values.  Fourth,  a  mathematical 
formulation  of  the  selective  maintenance  problem  was  developed.  Finally,  solution  procedures 
were  developed  to  solve  the  selective  maintenance  problem.  Enumerative  solution  strategies  for 
smaller  problems  and  heuristic  strategies  for  larger  ones  were  defined.  This  effort  clearly 
advanced  the  state  of  the  art  in  selective  maintenance  decision-making. 

6.  Quantification  of  Logistics  Capabilities 

In  2001,  the  Air  Force  Journal  of  Logistics  published  research  findings  by  Oliver,  et  al. 
[Oliver,  2001]  that  identified  the  key  logistics  and  operational  factors  associated  with  aircraft 
mission  capability  (MC).  In  their  research,  correlation  analysis  was  performed  to  identify  the 
key  factors  associated  with  MC  rates  and  various  logistical  factors  (such  as  logistics  functions 
and  personnel)  and  operational  factors  (such  as  funding  and  environment)  and  their  associated 
interactions.  Regression  analysis  was  used  to  explain  and  predict  F-16  MC  rates  using  quarterly 
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data  from  FY93-FY00.  Personnel  skill  levels,  cannibalization,  and  funding  levels  were  found  to 
be  the  most  significant  factors.  These  research  findings  led  to  the  recognition  that  the  USAF 
does  not  currently  have  a  metric  to  relate  maintenance  (MX)  personnel  skill  level  to  operational 
readiness. 

Building  upon  Oliver’s  work,  the  objectives  of  this  research  further  investigated  the 
relationship  between  personnel  skill  level  and  mission  capability  and  developed  an  associated 
metric  and  standard.  Specifically,  a  metric  that  measures  MC  rate  as  a  function  of  MX  personnel 
skill  levels  was  developed.  Once  the  metric  was  determined,  a  standard  for  that  metric  was 
identified  that  set  the  metric  value  that  the  Air  Force  should  strive  to  maintain  as  part  of  their 
operational  goals.  The  relationships  between  MX  personnel  skill  level  and  multiple  utilization 
and  reliability  and  maintainability  performance  measures  were  also  examined. 

The  research  methodology  consisted  of  performing  four  analysis  tasks  for  each  dependent 
variable.  The  first  task  was  to  define  how  the  variables  would  be  used  in  the  analysis.  Ten 
independent  variables  were  identified  to  MX  personnel  skill  level  including  the  count  and 
percentage  of  3-,  5-,  7-,  and  9-Level  maintainers.  The  dependent  variables  that  were  modeled 
included  MC  rate,  four  utilization  variables,  and  three  reliability  and  maintainability  variables. 
The  second  task  performed  a  correlation  analysis  between  the  dependent  and  independent 
variables.  Building  upon  the  second  task,  the  third  task  constructed  a  set  of  candidate  regression 
models  for  each  of  the  dependent  variables.  The  last  task  was  to  choose  a  final  model  for  each 
dependent  variable  by  examining  the  linear  fit  of  the  models,  the  efficiency  of  models,  and 
adherence  to  model  assumptions.  During  that  analysis  and  selection  process,  it  was  determined 
that  good  regression  models  for  flying  hours  and  sorties  (two  of  the  utilization  variables)  could 
not  be  developed  as  a  function  of  MX  personnel  skill  level  variables. 
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In  order  to  demonstrate  the  final  regression  models,  output  was  developed  into  a 
predicted  results  matrix  or  chart  to  show  the  effect  of  changes  in  personnel  levels  based  on  the 
dependent  variable  of  interest.  There  were  three  possible  representations  based  on  the  number  of 
variables  in  the  model: 

•  Single-variable  graph 

•  Dual-variable  matrix 

•  Triple-variable  series  of  matrices 

The  model  selected  for  MC  rate  contained  two  variables:  percentage  of  7-Level  and  9-Level 
maintained  (adjusted  R-Squared  value  of  80.7%).  Therefore,  the  recommended  metric  was  the 
percentage  of  7-Level  and  percentage  of  9-Level  maintained  employed.  Using  the  final  model 
for  MC  rate,  a  matrix  was  constructed  that  indicated  for  given  personnel  values,  whether  MC 
rates  could  be  expected  to  meet  or  exceed  standards  for  the  F-16C/D  aircraft. 

r 

A  software  tool  was  created  for  the  purpose  of  using  the  models  in  prediction  scenarios. 
The  tool  has  a  user  interface  that  allows  the  entry  of  possible  values  for  personnel  skill  and 
manning  levels.  Those  values  are  used  as  inputs  to  the  chosen  regression  models,  and  the  output 
for  each  performance  measure  is  computed  at  run  time.  The  tool  provides  an  example  of  the 
usefulness  of  the  regression  models  in  planning  situations. 

Although  limited  in  scope,  this  effort  certainly  lays  the  groundwork  for  follow-on 
research  into  a  perplexing  Air  Force  logistics  dilemma. 
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